Exploring Places¶
Experiments:¶
- Visualizing Places dataset
- Exploring Tags Places
- Exploring Towns & Places Names
- Exploring Properities
- Exploring Descriptions Places Similarities
- Descriptions Places Topic Modelling
1. Importing libraries and loading the json file with 5000 events to a dataframe¶
In [2]:
import json
import pandas as pd
import plotly.express as px
import os
import plotly.graph_objects as go
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
from bertopic import BERTopic
In [9]:
with open('places.json', 'r') as f:
data = json.load(f)
df = pd.DataFrame(data)
2. Visualizing the places dataframe¶
In [11]:
df.shape[0]
Out[11]:
Experiment 1: Exploring Place Ids¶
In [12]:
df_ids=df.groupby(['place_id']).size().reset_index()
df_ids=df_ids.rename(columns={0: "number_of_times"}).sort_values(by=['number_of_times'], ascending=False)
df_ids
Out[12]:
Experiment 2: Exploring Tags Places¶
We are going to separete the elements stored in each tag list into new rows.
In [13]:
df["tags"][0:5]
Out[13]:
In [14]:
df_tags=df.explode('tags')
In [15]:
df_tags
Out[15]:
In [16]:
g_tags=df_tags.groupby(['tags']).size().reset_index()
g_tags=g_tags.rename(columns={0: "number_of_times"}).sort_values(by=['number_of_times'], ascending=False)
g_tags
Out[16]:
In [17]:
px.histogram(g_tags, x="tags", y="number_of_times", histfunc="sum", color="tags", title='Frequency of tags places')
Experiment 3: Exploring Towns & Names¶
In [18]:
df["town"][1:10]
Out[18]:
3.1 Frequency of places grouped by towns¶
In [19]:
df_town=df.dropna(subset=['town'])
town=df_town.groupby(['town']).size().reset_index()
town=town.rename(columns={0: "number_of_times"})
town=town.drop([0])
In [20]:
town=town.sort_values(by=['number_of_times'], ascending=False)
town
Out[20]:
In [21]:
px.scatter(town, x='town', y='number_of_times', color='number_of_times', size="number_of_times", size_max=60, title="Frequency of places grouped by towns")
3.2 Frequency of places grouped by name¶
In [22]:
df_name_town=df.groupby(['name']).size().reset_index()
df_name_town=df_name_town.rename(columns={0: "number_of_times"})
df_name_town=df_name_town.sort_values(by=['number_of_times'], ascending=False)
df_name_town.reset_index()
Out[22]:
3.3. Frequency of places grouped by name and town¶
In [23]:
df_name_town=df.groupby(['name', 'town']).size().reset_index()
df_name_town=df_name_town.rename(columns={0: "number_of_times"})
df_name_town=df_name_town.sort_values(by=['number_of_times'], ascending=False)
df_name_town
Out[23]:
Experiment 4: Exploring Properities¶
In [24]:
df_properties=pd.concat([df.drop(['properties'], axis=1), df['properties'].apply(pd.Series)], axis=1)
In [25]:
df_properties[0:3]
Out[25]:
4.1 Frequency of places grouped by wheelchair-access and town¶
In [26]:
df_properties_wc=df_properties.groupby(['place.facilities.wheelchair-access', 'town']).size().reset_index()
df_properties_wc=df_properties_wc.rename(columns={0: "number_of_times"})
df_properties_wc=df_properties_wc.sort_values(by=['number_of_times'], ascending=False)
df_properties_wc
Out[26]:
4.2 Frequency of places grouped by toilets_disabled and town¶
5. Exploring Descriptions¶
In [31]:
df_descriptions=df.explode('descriptions')
df_descriptions=pd.concat([df_descriptions.drop(['descriptions'], axis=1), df_descriptions['descriptions'].apply(pd.Series)], axis=1)
df_descriptions=df_descriptions.dropna(subset=['description']).reset_index()
documents=df_descriptions["description"].values
In [32]:
len(documents)
Out[32]:
In [36]:
import re
from gensim.parsing.preprocessing import remove_stopwords
def clean_documents(text):
text = re.sub(r'\S*@\S*\s?', '', text, flags=re.MULTILINE) # remove email
text = re.sub(r'http\S+', '', text, flags=re.MULTILINE) # remove web addresses
text = re.sub("\'", "", text) # remove single quotes
text = remove_stopwords(text)
return text
In [37]:
d=[]
for text in documents:
d.append(clean_documents(text))
Generating Text Embeddings¶
In [38]:
model = SentenceTransformer('all-MiniLM-L6-v2')
#Training our text_embeddings - using the descriptions available & all-MiniLM-L6-v2 Transformer
text_embeddings = model.encode(documents, batch_size = 8, show_progress_bar = True)
In [39]:
np.shape(text_embeddings)
Out[39]:
Description Similarity¶
In [40]:
similarities = cosine_similarity(text_embeddings)
similarities_sorted = similarities.argsort()
id_1 = []
id_2 = []
score = []
for index,array in enumerate(similarities_sorted):
p=len(array)
id_1.append(index)
id_2.append(array[-2])
score.append(similarities[index][array[-2]])
index_df = pd.DataFrame({'id_1' : id_1,
'id_2' : id_2,
'score' : score})
print(index_df)
In [41]:
index_df["score"].sort_values(ascending=False)
Out[41]:
In [44]:
index_df.iloc[5]
Out[44]:
NOTE: Documents 5 and 7 seems to be the most similar. Lets see what they have
In [46]:
documents[5]
Out[46]:
In [47]:
documents[7]
Out[47]: